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ordering constraint, not restricted to epipolar lines. We prove several properties of the 
generalized ordering constraint and of the "forbidden zone", the set of matches that would 
violate the constraint. We consider both the orthographic and the perspective projection 
case, the latter for a simplified but standard stereo geometry. The disparity gradient limit 
found in the human stereo system may be related to a form of the ordering constraint. To 
illustrate our analysis we outline a simple algorithm that exploits the generalized ordering 
constraint for matching contours of wireframe objects. We also show that tire use of the 
generalized ordering constraint implies several other stereo matching constraints: a) the 
ordering constraint along epipolar lines, b) figural continuity, c) Binford’s cross-product 
constraint, d) Mayhew and Frisby’s figural continuity constraint. We finally discuss ways of 
extending the algorithm to arbitrary 3-D objects. 
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Figure 1 The geometry of the epipolar lines .The plane defined by the two foci, h\ and 
F 2 , and a point P cuts the two image planes in the corresponding epipolar lines 1 1 and l 2 . 
As we vary the position of P we produce a 1-parameter family of epipolar lines on each 
screen. The lines are not parallel to one another, and it can be seen that the epipolar lines 
on screen 1 all intersect at the point where the line FiF 2 hits screen 1. 


1. Introduction 

The problem of stereo matching is ill-posed and underdetermined: constraints are needed 
(a) to make the solution unique and (b) to reduce the search problem among possible 
matches. 

Marr and Poggio (1979) originally identified two important constraints: (1) uniqueness, that 
is, an element in one image in general only corresponds with a single element in the other 
image, and (2) continuity, that is, stereo disparity varies smoothly almost everywhere in the 
image. These constraints are powerful because they do not depend on the specific properties 
of the scene but on general properties of the stereo geometry. Marr and Poggio (1979) 
proposed a stereo matching algorithm, further developed by Grimson (1981, 1984), which 
incorporates the uniqueness and continuity constraints to match zero-crossing descriptions 
computed at different scales. An ordering constraint along epipolar lines has been exploited, 
both implicitly and explicitly, in several computer algorithms for stereo matching, as a special 
instance of the continuity constraint. Epipolar lines in the two images are lines on which 
corresponding points lie. The projections of a point P in space lie on the plane defined 
by P and the two camera foci and, as a consequence, on the two lines defined by the 
intersection of this plane with the two image planes (see figure 1). 

This implies that the matching problem can be reduced to a one-dimensional search if the 
epipoiars are known. Most algorithms assume that the epipolar geometry is known (from a 
known camera geometry) and that the images are registered. Furthermore, the ordering of 
edges or other features is usually preserved by stereo projection along epipolar lines (that 
is, if feature A is to the left of feature B in the left stereo image, then this spatial relationship 
is maintained in the right stereo image). The ordering constraint along epipolar lines follows 
from the continuity of surfaces and the assumption of opacity. As originally suggested by 
Baker (1982) the ordering constraint is violated in situations such as figure 2. Recently, 
Verri (1984) has discussed the role of the "forbidden zone", where the ordering constraint 
is violated (Krol and van de Grind, 1982 first introduced the notion of forbidden zone). The 
forbidden zone associated with each point of the visible surface is a set of points in space 
that would have images violating the ordering constraint. If any point in the forbidden zone 
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Figure 2 The forbidden zone associated with point A is represented by the dashed region. 
Any point in this region such as B has projections which violate the ordering constraint 
relative to A 


would be connected to the first point by an opaque surface the two images would "see" 
uppusiie sides of the surface. 

This ordering constraint can be exploited to reduce the complexity of the search for matching 
features, and to eliminate false matches. Interestingly, there is preliminary evidence that the 
ordering contraint (and perhaps a stronger form of it) may be implemented in the human 
stereo system (Burt & Julesz 1980). The human system, however, must often cope with 
situations in which the images are not precisely registered. Furthermore, physical edges 
are inherently two-dimensional (a property that is not exploited by the epipolar ordering 
constraint). It is therefore natural to ask whether the ordering constraint can be generalized 
from epipolar lines 1 . More precisely, can an ordering constraint be formulated that is 
independent of the epipolar geometry? The simplest use of such a constraint would be 
when either the epipolar lines are unknown or their estimation is affected by errors. 

In this paper we show that it is indeed possible to generalize the ordering constraint. We 
also give an analytic definition of the forbidden zone and characterize its properties. We 
show then that the generalized ordering constraint implies several other constraints that 
have been exploited in stereo matching. An algorithm based on this constraint has not been 
implemented yet, but we discuss its advantages and its limitations. 

The plan of the paper is as follows. Throughout the paper, unless specifically stated, we 
consider a stereo geometry in which the two image planes have the same vertical unit 
vectors. This simple geometry is fully representative for most practical applications and it 
represents a good approximation for human binocular geometry (Longuet-Higgins, 1982). In 
section 2 we assume orthographic projection. We prove a simple relationship between the 
two images of a 3D curve that leads to a generalization of the standard ordering constraint. 
This relationship allows us to identify special points in the images that correspond uniquely 
to the same physical point in the object curve. The Generalized Ordering Constraint (GOC) 
implies several of the specific constraints listed by Baker et al. (1983) , Mayhew and Frisby 
(1981) (see their figural continuity constraint), and Ohta and Kanade (1983). The ordering 
constraint breaks down when the object curve enters the forbidden zone. We define the 

1 This question was first brought to our attention by Dr. V. Torre 
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forbidden zone geometrically and algebraically and discuss its properties. Section 3 shows 
that slightly weaker results can be derived for perspective projection (this includes the 
ordering constraint along epipolar lines as a very special case). These results again break 
down for the forbidden zone, which is more complex than in the orthographic case. In 
section 3, we characterize fully the forbidden zone for the perspective case and prove several 
results about it. The boundaries of the forbidden zone correspond to the so called Panum 
limiting case in the psychophysics of stereo, when one line of sight just grazes the surface. 
Burt and Julesz' results suggest that human stereovision is limited to a smaller disparity 
gradient. In section 4 we derive the equations for the physical surfaces corresponding to 
this disparity gradient limit. In section 5 we show how the generalized ordering constraint 
implies other stereo matching constraints. In section 6 we outline an algorithm for stereo 
based on matching contours. From a single contour the algorithm retrieves the viewing 
parameters and unambiguously matches points along the contour using the generalized 
ordering constraint. The constraint of figural continuity (Mayhew and Frisby, 1981) follows 
from the generalized ordering constraint and is implicitly implemented in our algorithm. In 
Appendix 2 we derive an explicit solution of the equations for the viewing parameters in the 
case of the stereo geometry suggested by Longuet-Fliggins, using only two points. 


2. The Ordering Constraint and the Forbidden Zone for Orthographic 
Projection 


We assume a stereo-imaging geometry of the type proposed by Lonyuet-Higgiris (1962) in 
which the planes of the horizontal meridians of the two eyes coincide. Hence, the relative 
orientation of the two eyes is defined by one parameter only. We also assume orthographic 
projection. We show that for any part of the object which does not lie in the forbidden zone, 
there is a simple relationship between the images of the object in the two eyes. This enables 
us to generalize the ordering constraint (Baker & Binford, 1981) and to identify features in 
the images that correspond to the same feature in the object. 

We define an orthonormal triad of vectors in each eye. By our restriction on the geometry, 
the vertical direction, k, is the same for each eye. The right eye has vectors i, j and k, 
where j is normal to the right image plane. Similarly the left eye has vectors i', j' and k, 
where j' is normal to the left image plane The convergence angle $ satisfies j • j' = cost? 
(see Figure 3). We define coordinates X, Y and if along the i, j and k axes respectively. 
Similarly, we let X' and Y' be coordinates along the i' and j' directions. Note that the 
origins of these systems of coordinates lies at the intersection of the two image planes and 
not at the focal points of the eyes. The following equations connect the triads of the two 
eyes: 

j • j ' = COS 1? 

i • i ' — cos i? 

i' • j = — siin? (2 ' l) 

i • j' — -t-sini? 

We consider now a curve in the right image plane, parameterized by a (which is not the arc 
length) and written as 

r«(«) = X(a)i + Z(a)k (2.2) 

Under the assumption of orthographic projection, equation (2.2) is the image of an object 
curve given by r(a) =.- X(.s)i + F(,s)j + Z(s) k, 
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Figure 3 See text. 


r L (s)=X'(a)i' + Z’(a)k (2.3) 

where 

X '(a) = r(s) • i' = X(s) cos d — y(s)sin d 

Z’{s)= r{a)-k = Z{a) ' 2 ' 4 ^ 

We partition the right image curve into intervals for which Z is a single valued function of 
X. These intervals are separated by points at which — 0. For each such interval X can 
be chosen as the parameter and we can write the projection (in the right image plane) as 


r*(X) = Xi + Z(X) k 

Identification of a in equation (2.3) with X yields (for the left image) 

. r = X'(X)i' + Z'{X)k 

with 


X '(X) = X cos d - r(X) sin d 
Z'{X) = Z(X) 


Let us now compute ^ from equation (2.4): 


Note that (see Figure 4) 


dX' 

dX 


— COS d — 


dY{X) 

dX 


sin 


dY cos d 
dX < sin d 


(2.5) 

( 2 . 6 ) 

(2.7) 

( 2 . 8 ) 

(2.9) 


is the condition that curve never enters the "forbidden zone". Thus, if the curve never 
enters the forbidden zone, i.e., it satisfies the condition (2.9), then 
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Figure 4 Line (a) has gradient smaller than cotd and therefore is not in the forbidden 
zone. Line (b) has gradient larger than cotti and lies in the forbidden zone. The two views 
are of "opposite" sides. 


^> 0 . ( 2 . 10 ) 

Let us consider now the slope of the curve in the two images for each partition. The chain 
rule yields 


a/j _ aZi UA /Oil) 

dX' dXdX' ( ' ’ 

Thus, if the curve is outside the forbidden zone, (a) and jff- have the same sign (because 
of equation (2.10). Moreover, (b) the zeros of jfifr correspond to zeros of j §-. Furthermore, 
(c) equation (2.10) gives an ordering constraint on matching corresponding points in the 
two curves, since within each partition ordered points on the right image curve correspond 
to points on the right image curve with the same order (because of monotonicity implied by 
equation (2.11)). 


The Forbidden Zone for Orthographic Projection 


We now define the forbidden zone and show that it occurs precisely where the ordering 
constraints break down. 
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Figure 5 Points Ai and A 2 are the images of the physical point A and have coordinates 
Xi and X 2 , respectively. 


Suppose the two screens, screen 1 and screen 2, are at an angle 0 to each other as 
illustrated in Figures 5 and 6. Consider two points A and A' with coordinates (Xj.Yj) and 
(X[,Y[) respectively relative to screen 1. They will be projected to points A t and A[ on 
screen 1 with OAj — Xi and OA[ — X[. They will be projected to points A 2 and A 2 on 
screen 2 with OA 2 = X 2 = X x cos0 + Y x sin 0 and OA 2 = X 2 — X( cos0 + Y( sin 0. The 
ordering constraint will hold, provided 

X[ - Xj >0, if and only ifX' - X 2 > 0 (2.12) 

We have 

X 2 -X 2 = {X[ - Xi) cos 0{ 1 + tan( ?} (2.13) 

x 1 — X! 

So a necessary and sufficient condition for (2.12) to hold is that 

1 + > 0 (2.14) 

Note that we have —tt/2 < 0 < n/2 and hence cos# is always positive. 

Equation (2.14) can be interpreted as a condition on the gradient m of the straight line 
joining A to A'. Substituting m for yizQ in (2.14) gives 

m > — cot# (2.15) 

which is the same as equation (2.9). 

It is easy to see from Figure 6 that if this condition is violated, screen 1 and screen 2 will 
see opposite sides of the line joining A to A'. This motivates the following definition of the 
forbidden zone: 

•A point A' is in the forbidden zone of a point A if, and only if, the two screens see 
opposite sides of the straight line joining A' to A. 

As we have shown above, the forbidden zone is precisely the region where the ordering 
constraint breaks down. 


6 



/““N 


POGGIO 


A 



Figure 6 See text. 
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Observe from Figure 6 that the definition of the forbidden zone is symmetric, that is, if A' 
is in the forbidden zone of A, then A' is in the forbidden zone of A. Note also that the 
forbidden zone of A depends only on the position of A and the angle 0 between the two 
screens. This will not be the case when we consider perspective projection. In Section 3, 
we prove some results about the forbidden zone for perspective projection, which will also 
apply to orthographic projection. 


/•“s 


3. Perspective Projection 


We now consider the ordering constraints for perspective projection. 

The geometry is summarized in Figure 7. The two screens have unit normals a t and a 2 
and focal length m. The foci have position vectors fi, and f 2 relative to the origin 0. An 
arbitrary point A in space has position vector X relative to 0. 

The centers of screen 1 and screen 2 are r? and r§, where 


r° == fj — moj 
r° == f 2 — ma 2 


The equations of the screens are 


(3.1) 


r • fl! — fi • «i - m, for screen 1 
r. a2 = f 2 . « 2 - m, for screen 2. 

Now consider the projection of a point X on screen 1. The line of projection is 

r(X) = X + X(fj - X) 


From (3.2), this hits screen 1 at 


(3.2) 


(3.3) 



X • + X(f i — X) • a! = f i • ai — m 


We solve (3.4) to obtain 


(3.4) 
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Figure 7 See text. 




X — 1 = 


—m 

(f i - X) • ai 


and so X is projected to a point r! with 


i-i = fj 


m(f J - X) 
(fl-Xj-a, 


(3.5) 


(3.6) 


To find the position X x of the image of X relative to the center of the screen, rather than 
relative to 0, we must subtract r? from rj. Hence, 


Xi = ri - r ? = ^~ yr-((( f i - X) • «,)«! - (fi - X)) 
Similarly, the projection on screen 2 is given by 


(3.7) 


X* = (f ' T^ ' }Tfl a ^( fa ~ X )' - ( f * - x )) (»-8) 

Now we restrict ourselves to the geometry used by Longuet-Higgins (1982). In this case, 
the vectors and f 2 are coplanar and perpendicular to a vector k that we take to 

be in the z-direction. We can choose the origin 0 and the x-axis so that 


fi = -f 2 =(/,<>, 0) 


We define angles <j> x and <£ 2 , such that 

«i = (cos sin ^i,0) 
« 2 = (cos 4>2, sin <t> 2 , 0), 

and two vectors and orthogonal to and r* 2 


(3.9) 


(3.10) 
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Figure 8 See text. 


2* = (sin 4>i, — cos0) - 311 - 

«2 = (sin <p 2 > ~ cos <j> 2 , 0) 

The vectors aj, and k form a right-handed triad for screen 1, as do a* 2 , « 2 and k for 
screen 2, 

Let the components of X be (X, Y, Z). its projection Xj on screen 1 is given by (3.7), and 
can be written 


Xi=(X 1 -al)al+(X 1 -k)k (3.12) 

Note Xi • aj = 0, since Xj lies on the screen. Thus we can take Xi = (Xj • «;) and 
Z x = (Xi • k) to be the cartesian coordinates of the image on screen 1. Substituting from 
(3.11), (3.9) and (3.7) gives 


Xi = 
- 

Similarly, we have 

X 2 = 
Z 2 = 


/ COS 4>1 - 

- X cos 4>i - 
mZ 

- Y sin <t>\ 

f COS 4> 1 - 

- X cos <j>i - 

- Y sin 4>\ 


— m 


/ cos <f >2 + X cos (j >2 + Y sin <^ 2 
— mZ 


{—/ sin <t>i + X sin 4>i — Y cos <f>i) 


(3.13) 


(/ sin <j >2 + X s'u\ <f >2 — Y cos ^ 2 ) 


(3.M) 


/ cos <j >2 + X cos <f >2 + Y sin 4>i 


It is well-known that there is an ordering constraint along epipolar lines, and with our 
geometry, one such line occurs when Z — 0. This is illustrated by Figure 8, where X and 
X' lie in the Z ~ 0 plane. The forbidden zone of X is the shaded region. Then, provided 
that X' lies outside the forbidden zone of X the images of X' on the two screens will be 
either both to the right of the image of X, as in Figure 8, or both to the left. 
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Figure 9 See text. 




Now observe that both Xi and X 2 in (3.13) and (3.14) are independent of Z. Thus, if we 
remove the restriction Z = 0, we obtain a similar ordering constraint independent of whether 
X and X' lie on the same epipolar line. Intuitively, if X is to the "right" of X' in space, then 
the projections of X will be to the “right" of the projections of X' on both screens. 

These results hold if, and only if, X' is not in the forbidden zone of X. The forbidden zone 
is complicated for perspective projections, and we discuss it in the next subsection. 

One obvious application of our generalized ordering constraint concerns two contours on 
the two screens. It implies that there is an ordered mapping between the two contours and 
that the points at the end of the contours, where f— 0, must be identified. (See Figure 
9.) The ordering constraint along epipolar lines arises as a special case when we take the 
contours to be the epipolar lines. 


The Forbidden Zone for Perspective Projection. 




Since the X-component on the screen is independent of the ^-component in space, (3.13) 
and (3.14), the shape of the forbidden zone will be independent of Z. 

From Figure 10, it is clear that B' is in the forbidden zone of A if and only if the two screens 
see opposite sides of the straight line joining A to B'. 

The structure of the forbidden zone is more complicated for perspective than for orthographic 
projection. For orthographic projection, the angle subtended by A to the two screens was 
a fixed angle 0, independent of the position of A. For perspective, this is no longer the 
case. In this section, we characterize the forbidden zone algebraically and prove various 
desirable properties about it. 

Let the permissive zone of A be the complement of the forbidden zone of A. If B is in the 
permissive zone of A to the left of A we will write B left *A. Similarly, B right *A means B 
is in the permissive zone of A to the right of A. 

It is clear from Figure 10 that B left *A if and only if the angles and V’a are both positive. 
This will be true if and only if the /^-component of the cross-product of the vectors 1<\A 
with F 2 B and l<\A with F, B are both positive (since the lines are coplanar, the sign of the 
/^-component is the sign of sin^ 2 and sin^i, respectively). 

The vectors F 2 A and F 2 B are written 


10 
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Figure 10 B' lies in the forbidden zone of A but B does not. 






EsA = {Xi + f,Yi,o) 
f 2 b = (x 2 + /,r 2 ,o) 

and their cross-product is 

EzA x EzR = {(*i + /)>2 - (X 2 + /)r,}k 
Thus the point B is to the left of the line f\A if 

(X, + f)Y 2 - (X 2 + f)Yy > 0 

and to the right if 

(Xy + f)Y % - (X 2 + f)Yy < 0. 
Similarly, B is to the left of FyA if 

(Xy - !)Y 2 - (X 2 - J)Yy > 0 

and to the right if 

(Xy - J)Y 2 - (X 2 - f)Yy < 0. 

So we can have B left *A if and only if both 

(Xy + f)Y 2 - (X 2 + f)Yy > 0 
(Xy -f)Y a - (Xt -f)Yy > 0 

and B right*A if and only if both 

(Xy 4 I)Y 2 - (X 2 + f)Yy < 0 
(*i - f)Y 2 - (X 2 - f)Yy < O’ 


(3.15) 

(3.16) . 

(3.17) 

(3.18) 

(3.19) 

(3.20) 

(3.21) 

(3.22) 
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Figure 11 The line AB projects to a line on screen 1 and a point on screen 2. 

Observe that if Y 2 > Y 1 then if B is to the left of F X A it is automatically to the left of F 2 A. 
If Y< < Yi, then if B is to the left of F 2 A it is automatically to the left of F X A. This is an 
alternative way of characterizing left * and right*. 

We now show that the forbidden zone has the desirable properties of symmetry and 
transitivity. 

For svmmetrv. we must show that B left*A is equivalent to A right*B. In other words, if B 
is in the permissive zone of A to the left, then yt is in the permissive zone of B to the right 
and conversely. 

The conditions for B left*A are given by (3.21). The condition for A right*B are, from (3.22) 

( X 2 + m - (X! + f)Y 2 <0 
(X 2 - f)Yi - (X, - f)Y 2 < 0 

The equation (3.21) and (3.23) are clearly equivalent and hence the result is proved. 

Now we consider transitivity. We want to show that if C left* 0 and B left*yt then C left*A 

C teft*/t means that F 2 C is to the left of F 2 B and l>\C is to the left of FJ). From B left+,4 
we have F 2 B to the left of F 2 A and 1<\B is to the left of F X A. So, F 2 C is to the left of F 2 B 
which is to the left of F 2 A and hence, F 2 C is to the left of F 2 A. Similarly, F X C is to the left 
of FiA. Thus C left*yl and the result holds. 


4. Limits to Fusion 


We have shown in the preceding sections that when a point B is inside the forbidden zone 
of a point A, different sides of the straight line joining A to B are projected to different 
screens. If B lies on the boundary of the forbidden zone of A then, as in Figure 11, one 
side of the line will be visible to one screen but the other screen will only see a point. This 
corresponds to Panum’s Limiting case. 

If the point B is moved to B' in the permissive region of A then the line All' can be seen by 
both screen 1 and screen 2 although the projection to screen 2 will be very foreshortened, 
it should in principle be possible to fuse features on the line AB 1 despite this foreshortening. 
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Figure 12 See text 
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In a recent paper, however, Burt and Julesz (1980) claim that there is a limit to the relative 
disparities in two screens that the human visual system can fuse. If this is so, then the 
boundaries of the forbidden zone may be a competence limit while the Burt-Julesz limit may 
be a performance limit. We have so far been assuming that the greatest slopes in space 
that we can see are bounded above by the forbidden zone and Burt-Julesz’ results could 
tell us that this limit cannot always be attained. To investigate this, we ask what surfaces in 
space correspond to a limit of disparity gradient. We will see that they are straight lines. 

First, assume the system is fixating at infinity so the two screens are parallel. We illustrate 
this case by figure (12). Now let A be a point in the center of the visual field with coordinate 
(0, a). It projects to points A t and A 2 on the screens. These have components (/(1 + &), -m) 
and (-/(i + ^), —m) respectively. Let the relative disparity limit be denoted by a (Burt-Julesz 
would set « = 3). The relative disparity of two points in space is defined as the ratio of 
the differences of their images in the two screens (see Fig. 12). So the fusional limit would 
occur for points Q and P on the two screens, where Q is a distance X to the left of Ai and 
P is qX to the left of A 2 . So we have 


OQ = (f(l + ™)-\ -m) 

771 

£ 2 £ = (-/(! + -)-oX,-m) 

a 


The line QF y is given by 


y — 


m 


(*-/) 


(4.2) 


where y. = mf/a, and the line PF 2 is 


y = 


m 
a\ + 


{x + /). 


These lines intersect at a point where 


(4.3) 
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Figure 13 The screens fixate at A with mergence angle 2$ 




A(i + «) 

2 fx + X(a — 1) 
2m/ 

~ 2/x + X(a - 1) 


(4.4) 


Letting X van/, we obtain a curve j 


through A which is the greatest s! 


UpU UUI I UepUl IV* *1*5 


to the Burt-Julesz limit. Eliminating X from (4.4) gives the curve 


«* = (^|)/(«-!/) (4-5) 

which is a straight line. Observe that the gradient of the curve depends only on a and /. 
Setting a = 3 gives a gradient of ff. 

We also perform the calculation when the visual system is fixating on A as in Figure 13. 

It is straightforward to modify the previous calculation by setting 


m' — m cos <t> 
X ' — X cos <f> 


(4.6) 


Note that in this case we have 


tan <!>=—. (4.7) 

a 

Since neither m or X appear in the final expression (4.5), the result is unaltered by the 
change in fixation point. As before, the gradient is proportional to f and by (4.7) it is hence 
proportional to cos <j>. 

5. The generalized ordering constraint and other constraints 

Arnold and Binford (1980) and Mayhew and Frisby (1981) suggested a figural continuity 
/•"v, constraint for stereo matching: disparity is usually continuous along contours. This figural 

continuity constraint can be powerful in practical implementations as demonstrated by 
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/"" v Grimson (1984) and Ohta and Kanade (1984). Our geometric model of matching implicitely 

implements figural continuity, because matching is performed along continuous contours 
under the constraint that the disparity gradient is not too large (i.e., the contour does not 
enter the forbidden zone). 

Binford (see Baker et al., 1983) suggested surface occlusion rules for making explicit opacity 
and non-opacity of surfaces. In particular the cross-product rule determines whether an 
hypothetized match between two corners lies in the forbidden zone. Again a rule such as 
this follows immediately from the generalized ordering constraint. 

6. Epipolar lines as a constraint for matching 

We now proceed to show that if the stereo geometry is known, perhaps by registering 
the screens from the image as in Appendix 2, then there is a simple relation between the 
epipolar lines on the two screens. A point on the left screen will lie on a unique epipolar 
line which will therefore correspond to a unique epipolar line in the right screen. 

We consider a general point in space ( X , Y, Z). On the right screen, this projects to ( X lt Zi) 
by equation 3.13, 


Xj = 




/ cos </>i — X cos <j>i — Y sin </>i 
mZ 


(—/ sin <j>i + X sin (j>i —Y cos ) 


/ cos^i — X cos <j>i — Y sin <j>i 
and on the left screen to ( X 2 ,Z 2 ) by (3.14), 


(G.l) 

(6.2) 


X 2 = 


— tl% 

7 -■, ; ■ y -7—~ 77 ~ t — r (/ sin 0 2 + X sin <j> 2 

f cos <p 2 + X cos <p 2 + Y am <p 2 

—mZ 

f cos (J> 2 + X cos <j >2 + Y sin <j> 2 


— y cos $2) 


(6.3) 

(6.4) 


To derive our relation, we use the four equations (6.1)-(6.4) to eliminate the unknowns X, 
Y and Z, which leaves us with a single equation relating X is X 2 , Zi and Z 2 . 

We eliminate Z by dividing (6.2) by (6.4) to obtain 


Z\ —{f cos (j) 2 + X cos<j >2 + Y ain<f> 2 } 
Z 2 f cos <bi — X cos 4>i — Y sin <j>i 

We rewrite (6.1) and (6.3) respectively as 

X{ — Xj cos 4>\ — m sin 4>i) 

+ Y {—Xisin cfri + mcos^t} 

+ Xi} cos<^>i + mf sin <^1 == 0 

and 

X { — X 2 cos <j >2 — m, sin <j >2 } 

+ Y {—X 2 sin <j >2 + m cos <^ 2 } 

— X 3 f cos (f >2 — m/ sin <^ 2 — 0 

We combine (6.6) and (6.7) into a matrix equation 


(6.5) 


(6.6) 


(6.7) 
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f—Xi cos^fj — msin^i 
l—X 2 cos <f >2 — m sin <^2 


—Xj sin + tocos^i\/X\ _ f—Xif co$4>i 

—X 2 sin <j> 2 + m cos 4“i)'-Y / \ X 2 / cos <^ 2 


—m/sin 
+to/ sin <j )2 


( 6 . 8 ) 


We invert the matrix to solve for X and Y obtaining ' ’ 


x — /{(^!^ ~~ m ‘ 2 ) si[i(</>i + 0 a) - m(X 1 + y 2 )cos(^t + (^2)} 
{XiX 2 + m 2 } sin(^>2 — <£i) + to(X 2 — Xi) cos (</> 2 — ^1) 


and 


(6.9) 


y 


-a/ - 


{XlX 2 COS <j>2 COS (j> 1 


. .... V 


We rewrite (6.5) as 


(Xt COS <f) | + Z2 cos, $2)1 + (X2 cos <^2 “ X cos<^)X 

+ (X 2 sin <f >2 — Zi sin ^> l )V = 0. 

and substitute (6.9) and (6.10) into (6.11). After some manipulation, this simplifies and we 
obtain the result 


i?!{ to sin <j >2 + X 2 cos cf> 2 } — X 2 {msin 4>\ + Xj cos^t} (6.12) 

Thus we see that if wo know the position on one screen, for example (Xi, Zi), wo find 
that the other position on the other screen, (X 2 ,Z 2 ), must lie on a straight line. This*is, of 
course, the standard epipolar line constraint. It is interesting to see that in our coordinate 
system, it can be written in such a simple form. This makes it straightforward to implement 
it in an algorithm. 


7. Outline of a contour based algorithm 

A simple example of an algorithm which indicates how the generalized ordering constraint 
can be exploited is as follows. 

Instead of using the epipolar lines to impose an ordering constraint, we use them as a 
consistency constraint. In conjunction with the ordering constraint along the contour, they 
determine stereo matching. This avoids wasteful scanning of all epipolar lines and ensures 
that computation is done only at places where it is necessary. It also automatically enforces 
figural continuity. There are clearly interesting effects when the contour we are matching 
runs along an epipolar line for some distance. Psychophysical experiments (Buelthoff and 
Poggio, pers. comm.) suggest that the human visual system also has problems in such 
cases. 

For a single contour the generalized ordering constraint tells us that the leftmost point 
and the rightmost point in the two images must correspond (see figure 9). These points can 
be used to "register" the viewing system, using the formulae in Appendix 2, and hence to 
determine impliciteiy the epipolar lines. Starting at one of these two points we move along 
the contours using the epipolar line constraint (6.12) to determine which points are to be 
matched. If more than one contour is present the generalized ordering constraint is used 
to decide which contours in the two images correspond (contours are also "ordered" in a 
similar way as points along a single contour are). In this case there will be many points in 
the two images which are known to correspond (two for each contour) and the registration 
process will hence be more robust (Mayhew and Longuet-Higgins, 1982). An algorithm of 
this type is currently under development with E. Tiffany. 
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8. Discussion 


A few remarks about photometric effects and perspective invariants are relevant at 
this stage. We have assumed that the contours in the image being matched correspond 
to a physical structure, such as a wire or a silhouette, in the object being viewed. If 
zero-crossings of the Laplacian of a Gaussian are used as the matching primitives we must 
be sure they correspond to a precise physical location in the object being viewed. It is 
encouraging that results by Yuille (1984) suggest that many zero-crossings may be due 
to significant changes in the geometry of the object which are almost independent of the 
viewing positions. As it stands now, the theory presented here is only valid for wire-frame 
objects. Occluding contours, though theoretically "wrong", may however be often used in 
small angle stereo (Grimson, 1981). The main challenge is to extend the matching scheme 
presented here to solid and textured surfaces. Several possibilities can be considered: a) 
fingerprints representations (Yuille and Poggio, 1983) may provide specific features to be 
used in the matching process; b) several functional measurements of the two images (Krass, 
1984) along the contour may be used to perform the matching. 

The generalized ordering constraint analysis is done under the assumption of a three- 
dimensional curve in space. The analysis may be applied to occluding contours, but we 
need estimates of the errors. We also need to identify which contours are occluding 
contours. One possibility is to use fingerprints (Yuille and Poggio, 1983). The idea would 
then be to identify separately zero crossings corresponding to occluding contours, zero 
crossings corresponding to step edges and zero crossings due to texture. Some of these 
zero-crossings could then be used for a matching scheme based on the generalized ordering 
constraint. Interpolation of the surface hetwee n matched 7 ero crossing could be performed 
under the constraint of matching measurements provided by a scheme similar to Kass’ 
(1984). His scheme may be reformulated to exploit a form of ordering constraint for speeding 
up the search. The main problem for a practical algorithm is obviously the stability of the 
contours to be matched between the two images. Geometric and photometric distortions 
are likely to present a hard problem. 

It is also possible to use perspective invariants to help match two contours which 
arise from the same physical location. Points on the two contours which correspond to the 
same invariant can be matched. A number of perspective invariants or "semi-invariants" 
are discussed by Yuille and Verri (1984). For example, the zeros and the discontinuities of 
curvature of a non-planar curve are preserved under perspective projection to the image 
plane, although the converse is not true. These "semi-invariants" can be used to find points 
on two contours which correspond. 

In summary, we have shown that, with the Longuet-Fiiggins (1982) geometry, a 
generalized ordering constraint holds provided the viewed object does not enter the 
forbidden zone. This constraint is not restricted to epipolar lines. We characterize the 
forbidden zone for orthographic and perspective projections and discuss experiments which 
suggest that the human visual system incorrectly interprets objects in this zone. Results 
by Burt and Julesz (1980) suggest that limits to fusion occur before this zone is entered. 
We use the generalized ordering constraint to propose an algorithm for stereo matching 
along contours using the epipolar lines as a consistency constraint. We use the generalized 
ordering constraint to determine points in the images which correspond and hence can 
be used to find the viewing parameters. The generalized ordering constraint provides an 
efficient way to match points by scanning only along the available contours instead of along 
all epipolar lines. It also automatically imposes a figural continuity constraint (Mayhew, 
1983). 
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/"* K Appendix 1: The Orthographic Limit of Perspective Projection 

From equation (3.7), we have the perspective projection of a point X on a screen with 
normal a u focus f t and focal length rn is given by xy-f where: 

— Ttl 

x i = (f7_Txy—- x ) - <( f i - x ) • ffita) (/i-i-i) 

The term inside the brackets corresponds to the orthographic projection of (f t - X) on 
screen 1. It is scaled by a term (f t - X) • a,. Provided we stay in a region of the image 
where this term only varies a little, orthographic projection will be a good approximation 
and the scaling will be constant. This will normally be the case when (f - X) and are 
almost parallel, since we have 


(f — X) • a x = |f — X| cost (A 1.2) 

where r is the angle between (f- X) and For r near zero, cos r is approximated by 
1 - r 2 /2 and will be insensitive to small changes in r. 

How good the orthographic approximation is in general will depend on the gradient of 
the surface being projected not being too large. 





Appendix 2: Solving for the Viewing Angles 

The projection Xj of a point X on screen 1 is given by equation (3.7), and we write it 
as 

Xi - ma, - ■ ■ m . . (X - U) (A.2.1) 

A — f i • a 

Alternately, we can take a point f\ on screen 1 and construct the line joining it to the 
focus (Ft). The point P t has position vector Zj relative to the origin X 0 of screen 1. From 
Figure (1) and Figure (8)we see that 

X 0 = ft - ma i (A.2.2) 

The gradient of the line from Pi to f\ is ma\ - Zi, and we can write the line as 

r(X) = fi — maj + Zt + X{maj — Zi} (A.2.3) 

We rewrite (A.2.3) as 

r(X) = fi+(X-l){ma 1 -Zi} (A.2.4) 

and we obtain a similar equation for screen 2; 

r (/<) — f‘2 + (f 1 — 1 ){r»« 2 — Z 2 } (yt.2.5) 
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These lines intersect at a point P where 

fi + (X — l){?na t - Z[} — f 2 + (fi - l){m « 2 — Z 2 } (/t.2.6) 

Now in our geometry there is a fixed (vertical) axis k such that 

«i • k = « 2 • k = f t • k — f 2 • k = 0 (A.2.7) 

I\ lies on screen 1 so Z ( • = 0 and so we can define coordinates Zy{— Z x • k) and 

Xi of Zi by 




Zi = Zyk + Xy{k X ay) 


(A.2.8) 


Similarly, for Z 2 we have 


Z 2 — Z 2 k + X 2 (k x £s 2 ) 

Taking the dot product of (A.2.6) with k yields 

0 = -{,i - l )Z 2 + (X - 1 )Zi 

For convenience we replace X and n with f and v where 

S = X - 1 
v — fl — 1 

Then we rewrite (A.2.6) using (A.2.10) as 

fi — f2 = i^{ma 2 — Xi{k X a 2 )) 

- ({may -Xi(k X a x )) 

and equation (A.2.10) as 


£ _ £2 


Substituting for f from (A.2.13) into (A.2.12) gives 


(A. 2.9) 


(> 1 . 2 . 10 ) 


(A.2.11) 


(A.2.12) 


(A.2.13) 


fi - h — v{(ma. z - X 2 (k X a 2 )) ~ - -X"i(k X ax))} (A.2.14) 

_Now we assume a second point is also known to correspond. This has coefficients 
Zi,X 1 on screen 1 and Z 2 , 7 y 2 on screen 2. Repeating the argument leading up to (A.2.14) 
gives the equation 


fi~h — r {{ma 2 - X 2 (k X « 2 )) - —{may - X x(k X «x))} (A.2.15) 

A 1 

for some constant r. 

Define 2f = fi - f 2 and the angles 0 and <j> as in figure (14). Then taking the dot 
product of (A.2.15) with a x and n 2 yields 
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Figure 14 See text 


1 y?2 

—2/ cos(0 + <j>) = m cos 0 — X% sin 0 — —m 
v Z\ 


(A. 2.16) 


1 -^2 

—ccr. ^ coc 0 -j- Xi sin O'j 

IA Z\ 


Similarly from (A.2.15) we can obtain 


r A O Hi 

{J ,1>M> X t j 


l _ ~Z 2 

-2/ cos(0 + <j>) — mcosO — Xa sin# — zr"rn 
r 


(A2.18) 


1 

-2/ cos <b — m -(to cos 0 + Xj sin 0 ) 

r ~Z i 


Dividing (A.2.16) by (A.2.18) yields 


r rn cos 0 — X 2 sin 0 —-T^m 

v m cos 0 — X 2 sin 0 — m 

* 1 


Similarly, dividing (A.2.17) by (A.2.19) yields 


(A2.19) 


(A2.20) 


r to — §f(m cos 0 + Xy sin 0) 
y to — §*(m cos 0 + Xj sin 0) 

Hence we can combine (A.2.20) with (A.2.21) to yield an equation in 0 only. 


(A2.21) 


Z* ~Z 

(to cos 0 — Xo sin 0 — —to)(to — — (m cos (9 + sin 0)) — 

A 

^ Z 

(m cos 0 — X? sin 0 — —rn)(m — —-(to cos 0 + Xi sin 0)) 

Z\ Z\ 


(A.2.22) 
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O Set 


D = (>1.2.23) 

^ -h . 

Then, after some manipulation we write (A.2.22) as 

{rn 2 {A - B) + {AX l X 2 - BX 1 X 2 )} sin 2 # 

+ m{(X 2 -X 2 )+AB(X i -X l ))AnO (/1.2.24) 

+ m{B(X i — X 2 ) + A(X 2 — Xi)} sin # cos 0 = 0 

We can divide this equation by sin# ( sin# = 0 corresponds to the special case when 
the eyes are fixating at infinity) and obtain an equation of form 


C 1 sin # + C 2 + C 3 cos # — 0 


(A.2.25) 


where 




Cl = m 2 (A -B) + {AX iX 2 - BX\X 2 ) 

C 2 = m(X 2 - X 2 + AB(X t - XJ} (A.2.26) 

C 3 = m{B(X 1 - X 2 ) + >1(X 2 - Xi)} 

(A.2.25) can be written as a quadratic equation 

(C? + Cl) cos 2 # + 2C 2 C 3 cos # + (Cl - C 2 ) = 0 (A.2.27) 

and has solution 


-c 2 c 3 ± c.^/cf+cf^cl 

C? + C| 


(A.2.28) 


_ Note that (Ci,C 2 ,C 3 ) ~ (~C„-C 2) -C 3 ) as (A,73) ^ (/3,A) and (X,,X 2l Xi,X 2 ) w 
(Xi,X 2 ) X 1 ,X 2 ). So if we have two^points in the visual field which are known to coincide 
then we know that points Pi _and Pi on screenj correspond^ to P 2 and P 2 on screen 2. 
The coefficients (X„ Zi), (Y u Z{), (X 2 ,Z 2 ) and (X 2 ,Z 2 ) of P U P U P 2 and P 2 are known and 
hence from (A.2.23) and (A.2.26), C,,C 2 and C ;1 are known. (A.2.28) gives two possible 
solutions for # in terms of Ci,C 2 and C 3 . Let these be #1 and # 2 . Dividing (A.2.16) by 
(A.2.17) yields 


cos(# 4- <f) _ tocos# — X 2 sin # — Z 2 /Z \m 
cos<£ to — Z 2 j Zi{rn cos # + Xi siti #) 

Expanding cos(# + <j>) = cos#cos$ — sin#sin (j) we can rewrite (A.2.29) as 


where 


Ci sin <f> + C 2 cos <j> = 0 


~C\ = m — Z 2 jZ 1 (to cos # + Xi sin #) 

C 3 — —X 2 — Z 2 jZi{rn sin # — Xi cos #) 


(A.2.29) 


(A.2.30) 


(>1.2.31) 
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There are two possible values of {Ci,C 2 ,C 2 ) depending on whether 6 — 0 t or 0 — 0 2 . 
(A.2.31) will give two possible solutions for <j>, for each value of 0: 


COS 4 > — 


±Ci yjC 1 + C 3 

~ V\ + V\ 


(A.2.32) 


Hence we have four possible solutions for 0 and <f> which can be formed directly from 
(A.2.28) and (A.2.29). In the general case, only one of these solutions will be physically 
reasonable and the others can be discarded. 

We have shown that if the images on both screens of points in space are known then 
we can solve the non-linear equations directly for the angles 0 and <f> that specify the stereo 
geometry. These solutions are given by (A.2.28) and (A.2.32). Two solutions are generated 
of which only one is physically reasonable in general. 
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